Skip to content

feat(benchmarks): pnpm workspace + port SDK + switch consumers - #207

Merged
HeyGarrison merged 6 commits into
masterfrom
feat/bench-sdk-workspace
Jul 25, 2026
Merged

HeyGarrison merged 6 commits into
masterfrom
feat/bench-sdk-workspace

Conversation

@HeyGarrison

@HeyGarrison HeyGarrison commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Restructures the benchmarks repo to extract the bench SDK from the computesdk monorepo into a standalone publishable package, and consolidates all benchmark runners under a benchmarks/ wrapper.

SDK (publishable)

  • Extracted from @computesdk/bench in the computesdk monorepo
  • Lives at packages/benchsdk/ with npm name benchsdk (unscoped)
  • Built with tsup (CJS + ESM + DTS), tested with vitest (27 tests passing)
  • Version 0.2.0, ready for npm publish

Runners (in-house, under benchmarks/)

  • benchmarks/sandbox/ — sandbox TTI benchmarks (sequential, staggered, burst, dax)
  • benchmarks/browser/ — browser + throughput benchmarks
  • benchmarks/storage/ — storage + snapshot-fork benchmarks
  • benchmarks/ai-gateway/ — AI gateway benchmarks
  • benchmarks/scale/ — scale coordinator + Docker image
  • benchmarks/src/ — shared helpers (run.ts, ingest.ts, merge-results.ts, env.ts, update-readme.ts, util/)
  • benchmarks/scripts/ — operational scripts (backfill, build-e2b-template, dax-benchmark.sh, etc.)

Repo-level changes

  • Converted from npm to pnpm workspaces (pnpm-workspace.yaml declares packages/benchsdk)
  • Root package.json holds all runner scripts with benchmarks/-prefixed paths
  • All 8 CI workflows migrated to pnpm (pnpm install --frozen-lockfile, pnpm run bench)
  • @computesdk/bench dependency replaced with local benchsdk: workspace:*
  • Import specifiers in scale files rewritten from @computesdk/bench to benchsdk
  • .env.example moved to benchmarks/.env.example
  • Scale Dockerfile rewritten for pnpm + new directory structure

What stays at repo root

  • results/, *.svg, pricing.json — generated benchmark artifacts
  • README.md, LICENSE, CONTRIBUTING.md, METHODOLOGY.md, etc. — repo docs
  • packages/benchsdk/ — the SDK package

Rebased onto master

Includes all of master's latest providers (opencomputer, tilion, llmgateway, pydantic) and dependency bumps (beam, superserve, tenki, @codesandbox/sdk).

Commits

  1. fd90611 feat: convert benchmarks to pnpm workspaces with packages/bench + packages/cli
  2. 7ea9222 fix(cli): make bundle:scale paths relative to packages/cli cwd
  3. 66da0dd Update package.json (packageManager: pnpm@10.33.0)
  4. 9123781 chore(rebase): followup ai-gateway move after conflict resolution
  5. 2cfbc09 chore(rebase): sync pnpm-lock.yaml with post-rebase package.json
  6. 4d809af feat: restructure to benchsdk package + benchmarks/ wrapper

@open-cla

open-cla Bot commented Jul 22, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

View 2 additional findings in Devin Review.

Open in Devin Review

Comment thread package.json Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR converts the repo into a pnpm workspace with separate packages/bench (the benchmark orchestrator SDK) and packages/cli (the benchmark runner/consumer), then switches scale/storage/etc. CLI consumers from @computesdk/bench to the new local benchmarks workspace package and updates CI workflows accordingly.

Changes:

  • Introduces pnpm workspace layout (packages/bench, packages/cli) and migrates CI workflows to pnpm.
  • Ports the benchmark orchestrator client/worker helpers into packages/bench and updates CLI code to import from benchmarks.
  • Refactors benchmark runners (sandbox/storage/snapshot-fork/browser/scale) and adjusts result paths to the repo root results/ directory.

Reviewed changes

Copilot reviewed 44 out of 85 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/scale/Dockerfile Removes old root-level scale Dockerfile (moved under packages/cli).
pnpm-workspace.yaml Declares pnpm workspace packages (packages/bench, packages/cli).
packages/cli/tsconfig.json Adds CLI TypeScript configuration for the new workspace package.
packages/cli/src/util/timeout.ts Adds a shared promise timeout helper.
packages/cli/src/util/stats.ts Adds shared percentile/stats helpers (used by sandbox benchmarks).
packages/cli/src/util/error.ts Adds error formatting that preserves provider .cause chains.
packages/cli/src/update-readme.ts Updates repo-root path resolution for new workspace layout.
packages/cli/src/storage/types.ts Adds storage benchmark types and snapshot/fork override config shape.
packages/cli/src/storage/stats.ts Adds shared rounding/stats helpers for storage benchmarks.
packages/cli/src/storage/snapshot-fork-types.ts Introduces snapshot/fork dataset presets + result types.
packages/cli/src/storage/snapshot-fork-benchmark.ts Implements snapshot/fork benchmark runner + scoring + JSON writer.
packages/cli/src/storage/scoring.ts Adds composite scoring/sorting for storage benchmark results.
packages/cli/src/storage/providers.ts Defines storage provider configs using StorageSDK adapters.
packages/cli/src/storage/generate-svg.ts Updates repo-root path resolution for rendered storage SVGs.
packages/cli/src/storage/generate-snapshot-fork-svg.ts Updates repo-root path resolution for rendered snapshot/fork SVGs.
packages/cli/src/storage/benchmark.ts Implements upload/download/throughput storage benchmark + JSON writer.
packages/cli/src/scale/types.ts Adds types for scale/burst-100k result shapes and metrics.
packages/cli/src/scale/sinks/tigris.ts Adds Tigris S3-compatible sink for uploading raw/meta/metrics/logs.
packages/cli/src/scale/sdk-coordinator.ts Switches orchestrator client imports to benchmarks workspace package.
packages/cli/src/scale/scripts/watch.ts Switches orchestrator client imports to benchmarks.
packages/cli/src/scale/scripts/start.ts Switches orchestrator client imports to benchmarks.
packages/cli/src/scale/scripts/spinup-idle.ts Adds script to provision idle Namespace VMs for capacity dry-runs.
packages/cli/src/scale/scripts/kill.ts Adds script to list/destroy live Namespace VMs for a run prefix.
packages/cli/src/scale/scripts/aggregate.ts Switches orchestrator client imports to benchmarks.
packages/cli/src/scale/runner.ts Adds two-phase burst lifecycle runner + structured result emission.
packages/cli/src/scale/providers.ts Defines burst-100k provider participation and keep-alive settings.
packages/cli/src/scale/logger.ts Adds structured logger with in-memory transcript for uploads.
packages/cli/src/scale/Dockerfile.norun Adds a “bundle but don’t auto-run coordinator” scale image Dockerfile.
packages/cli/src/scale/Dockerfile Adds pnpm-workspace-aware scale image Dockerfile that bundles coordinator.
packages/cli/src/scale/bench-reporter.ts Switches orchestrator client import to benchmarks.
packages/cli/src/sandbox/types.ts Adds sandbox benchmark types (TTI, modes, results).
packages/cli/src/sandbox/table.ts Adds CLI table rendering + JSON writer for sandbox benchmark results.
packages/cli/src/sandbox/staggered.ts Adds staggered-mode sandbox benchmark runner.
packages/cli/src/sandbox/scoring.ts Adds sandbox composite scoring + sorting utilities.
packages/cli/src/sandbox/providers.ts Rehomes sandbox provider list under packages/cli workspace.
packages/cli/src/sandbox/generate-svg.ts Updates repo-root path resolution for sandbox SVG rendering.
packages/cli/src/sandbox/generate-pricing-svg.ts Updates repo-root path resolution for pricing SVG rendering.
packages/cli/src/sandbox/dax.ts Adds DAX workload benchmark runner using local dax-benchmark.sh.
packages/cli/src/sandbox/concurrent.ts Adds concurrent/burst-style sandbox benchmark runner.
packages/cli/src/sandbox/benchmark.ts Adds core sandbox benchmark iteration logic + reuse detection probe.
packages/cli/src/run.ts Updates results output directories to repo root and wires new modes.
packages/cli/src/merge-results.ts Updates repo-root path resolution for merging results under workspaces.
packages/cli/src/ingest.ts Updates repo-root path resolution for ingest tooling under workspaces.
packages/cli/src/env.ts Updates .env resolution to repo root under workspaces.
packages/cli/src/browser/types.ts Adds browser benchmark types.
packages/cli/src/browser/throughput-types.ts Adds browser throughput benchmark types/constants.
packages/cli/src/browser/throughput-scoring.ts Adds throughput composite scoring + sorting.
packages/cli/src/browser/throughput-providers.ts Adds throughput provider configs with standardized viewport options.
packages/cli/src/browser/scoring.ts Adds browser benchmark composite scoring + sorting.
packages/cli/src/browser/providers.ts Adds browser provider configs.
packages/cli/src/browser/generate-throughput-svg.ts Updates repo-root path resolution for throughput SVG rendering.
packages/cli/src/browser/generate-svg.ts Updates repo-root path resolution for browser SVG rendering.
packages/cli/src/browser/benchmark.ts Adds browser benchmark runner using Playwright CDP connection.
packages/cli/scripts/find-northflank-plan.ts Adds helper script to select a suitable Northflank deployment plan.
packages/cli/scripts/dax-benchmark.sh Adds local bash script for DAX clone/install/typecheck workload.
packages/cli/scripts/build-e2b-template.ts Adds helper to build an E2B template with fixed resources.
packages/cli/scripts/backfill.ts Updates repo-root path resolution for backfill tooling under workspaces.
packages/cli/package.json Adds CLI package scripts and depends on benchmarks workspace package.
packages/cli/env.example Adds example environment variables for running benchmarks locally.
packages/bench/vitest.config.ts Adds Vitest config for the benchmarks SDK package.
packages/bench/tsup.config.ts Adds tsup build config for publishing benchmarks SDK.
packages/bench/tsconfig.json Adds package tsconfig extending repo root config.
packages/bench/src/reporter.ts Adds BenchmarkReporter wrapper for best-effort telemetry/reporting.
packages/bench/src/metrics.ts Adds system metrics sampler for coordinators/workers.
packages/bench/src/index.ts Exports client/worker helpers, reporter, metrics, and public types.
packages/bench/src/tests/integration.test.ts Adds optional live integration smoke test (env-gated).
packages/bench/README.md Adds usage/API docs for the new benchmarks SDK package.
packages/bench/package.json Defines publishable benchmarks SDK package metadata and scripts.
package.json Removes root scripts and drops @computesdk/bench dependency in favor of workspace package.
.npmrc Adds pnpm setting (shamefully-hoist=true) for workspace installs.
.github/workflows/storage-benchmarks.yml Migrates storage benchmark workflow to pnpm + new paths.
.github/workflows/snapshot-fork-benchmarks.yml Migrates snapshot/fork workflow to pnpm + new paths.
.github/workflows/scale-image.yml Updates scale image workflow paths to packages/cli/src/scale/**.
.github/workflows/sandbox-tti-benchmarks.yml Migrates sandbox TTI workflow to pnpm + new paths.
.github/workflows/sandbox-dax-benchmarks.yml Migrates DAX workflow to pnpm + new paths.
.github/workflows/pricing-svg.yml Migrates pricing SVG workflow to pnpm.
.github/workflows/browser-throughput-benchmarks.yml Migrates browser throughput workflow to pnpm + new paths.
.github/workflows/browser-benchmarks.yml Migrates browser benchmark workflow to pnpm + new paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@HeyGarrison
HeyGarrison force-pushed the feat/bench-sdk-workspace branch 2 times, most recently from b059020 to 255d621 Compare July 22, 2026 21:05

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 1 security concern(s).

Comment thread .github/workflows/pricing-svg.yml Resolved
@superagent-security superagent-security Bot added pr:flagged PR flagged for review by security analysis. and removed pr:flagged PR flagged for review by security analysis. labels Jul 22, 2026
HeyGarrison and others added 5 commits July 24, 2026 20:27
…kages/cli

Port the @computesdk/bench SDK verbatim into packages/bench (published name
'benchmarks', v0.2.0) and move the CLI runner tree into packages/cli
(@benchmarks/cli, private). Add pnpm-workspace.yaml, replace package-lock.json
with pnpm-lock.yaml, adjust __dirname/ROOT path math in packages/cli/src to
resolve to the repo root, and move the @computesdk/bench dependency to
packages/cli as benchmarks: workspace:*. results/ stays at repo root.
The bundle:scale script used repo-root-relative paths that only resolved
from repo root, breaking 'pnpm --filter @benchmarks/cli run bundle:scale'.
Use paths relative to the package cwd so it works via workspace filter;
bundle:burst-100k delegates to it. The Dockerfile keeps its own inline
repo-root esbuild command for backwards compatibility.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Move src/ai-gateway/* to packages/cli/src/ai-gateway/* (master added this tree
  post-F2-cherry-pick)
- Add 6 master-added bench:ai-gateway:* + generate-ai-gateway-svg scripts to
  packages/cli/package.json with paths rewritten to packages/cli/src/...
- Empty root package.json scripts per F2 intent (bench:* / generate:* /
  bundle:* / update-readme all live in packages/cli/package.json now)
- Adjust __dirname/.. path math in packages/cli/src/ai-gateway/generate-svg.ts
  (was '../..' at src/, becomes '../../..' one level deeper under packages/cli/)
- Path-fix 62 cli scripts that referenced src/... instead of packages/cli/src/...
- Remove @computesdk/bench from root deps (validation contract expects it
  only in packages/cli/package.json as benchmarks: workspace:*)
- Add benchmarks: workspace:* to packages/cli/package.json dependencies
- Bump @computesdk/createos-sandbox 0.1.3 -> 0.1.4 (master updated)
- Bump @computesdk/upstash 0.3.7 -> 0.3.9 (master updated)
- Drop @changesets/cli from root devDeps (was lifted from packages/cli via shamefully-hoist)
@HeyGarrison
HeyGarrison force-pushed the feat/bench-sdk-workspace branch 2 times, most recently from 2fa0d70 to 24ad21e Compare July 24, 2026 21:19
@github-actions

github-actions Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

AI Gateway Benchmark Results

# Gateway Score Cold E2E Warm TTFT Tok/sec Status
1 Cloudflare AI Gateway 92.3 624ms 452ms 99.4 4/4
2 Anthropic (direct) 90.9 984ms 524ms 92.8 4/4
3 LLM Gateway 90.6 953ms 651ms 91.7 4/4
4 OpenRouter 90.1 1348ms 690ms 104.8 4/4
5 Vercel AI Gateway 88.5 1745ms 548ms 90.0 4/4
6 Pydantic AI Gateway 88.4 1726ms 551ms 89.8 4/4

View full run · SVG available as a build artifact

@HeyGarrison
HeyGarrison force-pushed the feat/bench-sdk-workspace branch from 24ad21e to 4d809af Compare July 24, 2026 21:21
with:
# Full history so the rebase-on-push retry below has a merge base.
fetch-depth: 0
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is compromised.

Pin the action to a full commit SHA and verify it against the official repository.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/ai-gateway-benchmarks.yml">
<violation number="1" location=".github/workflows/ai-gateway-benchmarks.yml:56">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>Uses `pnpm/action-setup@v4` with a mutable version tag instead of a commit SHA in `.github/workflows/ai-gateway-benchmarks.yml:56`.</evidence>
<recommendation>Pin the action to a full commit SHA (e.g., `pnpm/action-setup@&lt;sha&gt;`) and verify it against the official pnpm/action-setup repository.</recommendation>
</violation>
</file>

@superagent-security superagent-security Bot added the pr:flagged PR flagged for review by security analysis. label Jul 24, 2026
@HeyGarrison

Copy link
Copy Markdown
Collaborator Author

@devin can you review this?

@devin-ai-integration

Copy link
Copy Markdown
Contributor

View Devin session

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Review: pnpm workspace + benchsdk port + benchmarks/ wrapper

I checked out feat/bench-sdk-workspace, ran pnpm install --frozen-lockfile, built and tested benchsdk, ran the root typecheck, and diffed every moved file against master.

The good news first: the move itself is faithful. All 20 moved .ts files that show up as modified differ from master only in import specifiers / __dirname depth (including dax.ts's BENCH_SCRIPT_PATH) — no logic drifted. benchsdk's generated dist/index.d.ts is byte-identical to the published @computesdk/bench@0.1.9, its 27 tests pass, its own tsc --noEmit is clean, and the lockfile is in sync.

But there are four things that will break on merge.


1. 🔴 All committed benchmark history got nested one level deeper: results/ → results/results/

4d809af moved 661 result files (every mode except ai-gateway, plus results/.gitkeep) into results/results/:

R100  results/sequential_tti/latest.json  →  results/results/sequential_tti/latest.json
R100  results/storage/1mb/latest.json     →  results/results/storage/1mb/latest.json
...661 files

Every reader still points at <repo>/results/<mode> (benchmarks/*/generate-svg.ts, benchmarks/src/{run,merge-results,ingest,update-readme}.ts all resolve ROOT = __dirname/../..), so the whole SVG/README/ingest pipeline sees an empty history. Reproduced on both branches:

# this branch
$ pnpm run generate-svg
No results found for mode: sequential / staggered / burst
No benchmark results found for any mode → exit 1

# master
$ npx tsx src/sandbox/generate-svg.ts
SVG written to sequential_tti.svg … staggered_tti.svg … burst_tti.svg → exit 0

Worth noting master's HEAD is literally cc89155 "restores deleted results and fixes recurring issue" — this is the same class of regression. Fix is a straight git mv results/results/* results/ && rmdir results/results.

2. 🔴 7 of the 8 workflows switched to pnpm but lost the pnpm/action-setup step

fd90611 added - uses: pnpm/action-setup@v4 to every workflow; the restructure commit 4d809af appears to have regenerated the workflows from master's npm versions and only re-added it to ai-gateway-benchmarks.yml:56. Every other workflow now has cache: 'pnpm' + pnpm install --frozen-lockfile with no pnpm on the runner:

workflow cache: 'pnpm' pnpm/action-setup
ai-gateway-benchmarks.yml ✅ ✅
browser-benchmarks.yml 2 jobs ❌
browser-throughput-benchmarks.yml 2 jobs ❌
pricing-svg.yml ✅ ❌
sandbox-dax-benchmarks.yml 2 jobs ❌
sandbox-tti-benchmarks.yml 2 jobs ❌
snapshot-fork-benchmarks.yml 2 jobs ❌
storage-benchmarks.yml 2 jobs ❌

The runner image ships npm and Yarn but not pnpm, so actions/setup-node fails at cache: 'pnpm' with Unable to locate executable file: pnpm before anything installs. None of these ran on this PR (their paths: filters don't match a workflow-only change on a PR), so green CI here isn't evidence. When re-adding, pin to a SHA — that's also what the currently-failing Superagent Security Scan is complaining about on the one workflow that does have it.

3. 🔴 benchsdk is never built, so a fresh clone can't run or typecheck the scale runners

packages/benchsdk has build/clean/dev but no prepare/postinstall, so after pnpm install there is no packages/benchsdk/dist, and main/types point into it. Straight from a clean install:

$ pnpm run typecheck
benchmarks/scale/scripts/start.ts(46,39): error TS2307: Cannot find module 'benchsdk' …
benchmarks/scale/scripts/watch.ts(31,58): error TS2307: …
benchmarks/scale/sdk-coordinator.ts(5,83): error TS2307: …
(+18 cascading implicit-any errors)   → 22 errors

$ pnpm --filter benchsdk run build && pnpm run typecheck
benchmarks/scripts/build-e2b-template.ts(2,46): error TS2307: Cannot find module 'e2b'  → 1 error

Same for runtime: bench:scale:local / :start / :watch / :aggregate all import benchsdk under tsx. The Docker image is fine only because it explicitly runs pnpm --filter benchsdk run build. Adding "prepare": "tsup" to packages/benchsdk/package.json fixes both.

(That last e2b error is newly surfaced rather than newly introduced — the new root tsconfig.json adds benchmarks/scripts/** to include, and scripts/build-e2b-template.ts imports e2b, which isn't a dependency. Worth adding the dep or excluding the file, since the new typecheck script can't pass as-is.)

4. 🔴 The rebase dropped five scripts that exist on master

master's package.json has these; the branch's rewritten script block doesn't:

bench:ai-gateway:llmgateway
bench:ai-gateway:pydantic
bench:browser:tilion
bench:browser-throughput:tilion
bench:opencomputer

All five providers still exist in code (benchmarks/ai-gateway/providers.ts:54,76, benchmarks/browser/providers.ts:77, benchmarks/browser/throughput-providers.ts:103, benchmarks/sandbox/providers.ts:155) and AI_GATEWAYS.md:117-118 documents the two gateway ones.


Should fix before merge

  • packages/benchsdk/README.md is still the @computesdk/bench README — title, npm install @computesdk/bench, and every import example. It's in files, so the published benchsdk tarball would tell users to install a different package. package.json also has a stale repository.directory: "packages/bench".
  • benchmarks/scale/Dockerfile.norun is dead — it still does COPY package.json package-lock.json ./, npm ci, COPY src ./src; the lockfile and src/ are both gone in this PR.
  • .env location changed silently. benchmarks/src/env.ts loads ../.env, i.e. benchmarks/.env now, and env.example moved to benchmarks/.env.example. CONTRIBUTING.md:72 and METHODOLOGY.md:305 still say cp env.example .env at the root — existing local .env files stop being picked up with no error. Same staleness in AI_GATEWAYS.md (src/ai-gateway/…, npx tsx src/run.ts) and .claude/skills/add-sandbox-provider/SKILL.md (links to src/sandbox/providers.ts, npm install), plus all the npm run snippets across the docs.
  • The new package has no CI. Nothing builds, typechecks, tests or publishes benchsdk on PRs — for a package that's meant to go to npm, a small workflow running pnpm --filter benchsdk run build && test && typecheck seems worth it. Also "lint": "eslint" with no eslint config anywhere in the repo will fail if anyone runs it, and the devDeps are skewed from the root (vitest 1.x, eslint 8 EOL, TS ^5 vs root TS ^6).

Nits

  • __BENCH_VERSION__ is defined in tsup.config.ts but never referenced in src/.
  • pnpm blocks postinstall scripts by default — installs print Ignored build scripts: esbuild, protobufjs, bufferutil, cbor-extract, …. Nothing broke in my testing, but an explicit pnpm.onlyBuiltDependencies in the root package.json would make that deliberate rather than a warning everyone learns to ignore.
  • Workflows still shell out to npx tsx … in a pnpm repo; pnpm exec tsx is more consistent (and avoids npx's fallback-to-registry behaviour).
  • benchsdk is unscoped and currently unpublished on npm, while everything else here is @computesdk/*. If unscoped is deliberate, fine — otherwise @computesdk/benchsdk is more consistent and not squattable. Either way, the existing @computesdk/bench@0.1.9 probably wants a deprecation pointing at the new name.

@HeyGarrison
HeyGarrison force-pushed the feat/bench-sdk-workspace branch from 4d809af to abe635f Compare July 25, 2026 12:18
@devin-ai-integration

Copy link
Copy Markdown
Contributor

Re-review of abe635f

All four blockers are genuinely fixed — verified by clean clone + pnpm install --frozen-lockfile:

# Was Now
1 661 results nested in results/results/ git diff --merge-base master -- results/ is empty — history restored byte-for-byte. pnpm run generate-svg writes all four SVGs and leaves the tree clean
2 pnpm/action-setup missing in 7 workflows present in all 8, immediately before actions/setup-node in every job
3 benchsdk never built "prepare": "tsup" → dist/ exists straight after install; pnpm run typecheck is now completely clean (e2b added as a dep, so the old TS2307 is gone too); 27 benchsdk tests pass
4 5 scripts lost in the rebase all restored — typecheck is the only script that differs from master now

The smaller items are done too: benchsdk README and repository.directory renamed, Dockerfile.norun deleted, docs moved to pnpm + benchmarks/.env.example.


🔴 One new regression: the prepare script broke the scale image build

Build and push scale image is now failing on this PR:

#12 2.031 packages/benchsdk prepare$ tsup
#12 2.196 packages/benchsdk prepare: No input files, try "tsup <your-file>" instead
#12 2.200 packages/benchsdk prepare: Failed
ERROR: process "/bin/sh -c pnpm install --frozen-lockfile" did not complete successfully: exit code: 1

benchmarks/scale/Dockerfile copies only the manifests first (for layer caching), so pnpm install now runs prepare → tsup before packages/benchsdk/src has been copied in:

COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
COPY packages/benchsdk/package.json packages/benchsdk/
COPY tsconfig.json ./
RUN pnpm install --frozen-lockfile      # ← prepare runs here, no src yet
COPY benchmarks ./benchmarks
COPY packages/benchsdk ./packages/benchsdk
RUN pnpm --filter benchsdk run build    # ← the real build, already here

Smallest fix, which I built locally to confirm (docker build -f benchmarks/scale/Dockerfile . fails on abe635f, succeeds with this one-word change):

-RUN pnpm install --frozen-lockfile
+RUN pnpm install --frozen-lockfile --ignore-scripts

The explicit pnpm --filter benchsdk run build after the COPY still produces dist/, so nothing else changes.

Still open (all minor, none blocking)

  • Superagent Security Scan is still red: pnpm/action-setup@v4 is unpinned, now in 14 places. Pinning to a SHA (pnpm/action-setup@<sha> # v4.x.x) clears it.
  • .claude/skills/add-sandbox-provider/SKILL.md wasn't updated — it still tells the next agent to edit env.example, src/sandbox/providers.ts, src/sandbox/generate-svg.ts, src/sandbox/benchmark.ts and run npm install. Since that skill is the documented path for adding providers, it'll produce wrong edits on the next provider PR.
  • CONTRIBUTING.md:71 and METHODOLOGY.md:304 still say npm install directly above the pnpm run … blocks; AI_GATEWAYS.md:128 still cites env.example.
  • benchsdk still has "lint": "eslint" with no eslint config anywhere in the repo, and no workflow builds/tests/publishes it on PRs.
  • __BENCH_VERSION__ is still defined in tsup.config.ts and never referenced.
  • e2b: ^2.35.0 landed out of alphabetical order (right after @computesdk/e2b) in dependencies.

Fix the Dockerfile line and this looks good to merge from my side.

- Rename SDK from packages/bench to packages/benchsdk (npm name: benchsdk)
- Move all runners, src, scripts, results, SVGs, pricing into benchmarks/
- Update all __dirname path math for new directory depth
- Rewrite SDK imports from @computesdk/bench to benchsdk
- Update 8 CI workflows: benchmarks/ paths + pnpm commands
- Fix relative imports: ./sandbox/ -> ../sandbox/, ../util/ -> ../src/util/
- Delete packages/cli (scripts consolidated into root package.json)
- Add pnpm/action-setup to ai-gateway workflow
- Rebase onto origin/master: merge master's new providers (opencomputer,
  tilion, llmgateway, pydantic) and dep bumps (beam, superserve, tenki)
@HeyGarrison
HeyGarrison force-pushed the feat/bench-sdk-workspace branch from abe635f to ec8a8c4 Compare July 25, 2026 12:57
@superagent-security superagent-security Bot removed the pr:flagged PR flagged for review by security analysis. label Jul 25, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

Re-review of ec8a8c4 — LGTM

The Dockerfile regression is fixed and everything I flagged is now clean. Verified from a fresh node_modules:

  • pnpm install --frozen-lockfile → packages/benchsdk/dist present
  • pnpm run typecheck → clean
  • pnpm --filter benchsdk test → 27 passed, 1 skipped
  • pnpm run generate-svg → all four SVGs regenerate identically (no tree diff), git diff --merge-base master -- results/ still empty
  • docker build -f benchmarks/scale/Dockerfile . → builds locally, and Build and push scale image plus Superagent Security Scan are both green on CI now (5/5 passing)

Also confirmed the doc/skill cleanup: .claude/skills/add-sandbox-provider/SKILL.md now points at benchmarks/.env.example + benchmarks/sandbox/*, npm install → pnpm install in CONTRIBUTING/METHODOLOGY, and e2b moved into alphabetical order.

Only leftovers are cosmetic and fine as follow-ups: benchsdk still has "lint": "eslint" with no eslint config, nothing in CI builds/tests the package on PRs, and __BENCH_VERSION__ is defined in tsup.config.ts but never referenced.

Ship it.

@HeyGarrison
HeyGarrison merged commit 51e918c into master Jul 25, 2026
5 checks passed
@HeyGarrison
HeyGarrison deleted the feat/bench-sdk-workspace branch July 25, 2026 13:18
@github-actions

Copy link
Copy Markdown
Contributor

Storage Benchmark Results

1MB Files

# Provider Score Download Throughput Upload Status
1 Azure Blob Storage 96.2 0.03s 297.0 Mbps 0.10s 10/10
2 Tigris 94.9 0.05s 185.3 Mbps 0.57s 10/10
3 Vercel Blob 94.5 0.17s 50.8 Mbps 0.24s 10/10
4 AWS S3 94.5 0.16s 53.8 Mbps 0.32s 10/10
5 Cloudflare R2 94.2 0.16s 52.7 Mbps 0.50s 10/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Snapshot/Fork Benchmark Results

small dataset

# Provider Score Snapshot create Fork (snapshot) Fork (live) First read Status
1 Tigris 99.6 0.14s 0.41s 0.26s 0.25s 1/1
2 Azure-blob 98.8 0.59s 1.06s 0.60s 0.02s 1/1
3 Cloudflare R2 95.8 2.20s 2.70s 4.72s 0.09s 1/1
4 AWS S3 0.0 0.00s 0.00s 0.00s 0.00s 0/1

View full run

@github-actions

Copy link
Copy Markdown
Contributor

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Kernel 97.9 0.03s 0.05s 0.11s 0.05s 0.24s 10/10
2 Tilion 96.9 0.02s 0.04s 0.09s 0.01s 0.17s 10/10
3 Browseruse 94.3 0.27s 0.17s 0.09s 0.06s 0.64s 10/10
4 Browserbase 91.6 0.22s 0.12s 0.13s 0.13s 0.61s 10/10
5 Notte 90.5 0.20s 0.50s 0.19s 0.09s 1.05s 10/10
6 Hyperbrowser 88.4 0.29s 0.54s 0.24s 0.10s 1.24s 10/10
7 Steel 57.5 0.26s 1.33s 0.14s 0.17s 2.93s 10/10

View full run · SVG available as build artifact

@github-actions

Copy link
Copy Markdown
Contributor

Browser Throughput Benchmark Results

# Provider Score APS (med) Task (med) Task (p95) Screenshot Status
1 Kernel 73.2 4.24/s 2.36s 2.53s 237ms 3/3
2 Tilion 64.8 2.70/s 3.70s 4.17s 366ms 3/3
3 Browseruse 63.7 2.49/s 4.02s 4.15s 362ms 3/3
4 Browserbase 59.0 1.98/s 5.05s 6.79s 371ms 3/3
5 Steel 55.9 1.57/s 6.38s 7.11s 663ms 3/3
6 Hyperbrowser 51.2 1.17/s 8.54s 8.80s 922ms 3/3
7 Notte 44.0 0.93/s 10.77s 15.38s 938ms 3/3

View full run · SVG available as build artifact

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 isorun 99.1 0.01s 0.21s 0.21s 10/10
2 createos 98.7 0.05s 0.24s 0.24s 10/10
3 declaw 98.5 0.05s 0.29s 0.29s 10/10
4 northflank 98.5 0.09s 0.24s 0.24s 10/10
5 lightning 97.4 0.16s 0.42s 0.42s 10/10
6 blaxel 96.9 0.25s 0.41s 0.41s 10/10
7 vercel 96.3 0.29s 0.47s 0.47s 10/10
8 upstash 96.2 0.30s 0.50s 0.50s 10/10
9 superserve 96.2 0.33s 0.46s 0.46s 10/10
10 modal 94.6 0.45s 0.66s 0.66s 10/10
11 archil 94.4 0.13s 1.21s 1.21s 10/10
12 runloop 92.8 0.48s 1.07s 1.07s 10/10
13 tensorlake 92.7 0.70s 0.77s 0.77s 10/10
14 daytona 91.3 0.69s 1.15s 1.15s 10/10
15 tenki 89.4 0.90s 1.31s 1.31s 10/10
16 beam 87.3 0.19s 2.88s 2.88s 10/10
17 e2b 86.5 1.11s 1.72s 1.72s 10/10
18 cloud-run 80.2 0.60s 4.05s 4.05s 10/10
19 codesandbox 73.7 2.49s 2.84s 2.84s 10/10
20 opencomputer 63.3 1.95s 6.24s 6.24s 10/10
21 cloudflare 61.5 3.07s 5.02s 5.02s 10/10
22 hopx 0.0 0.00s 0.00s 0.00s 0/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.01s 0.04s 0.04s 10/10
2 createos 99.5 0.05s 0.06s 0.06s 10/10
3 declaw 99.4 0.05s 0.07s 0.07s 10/10
4 northflank 99.1 0.07s 0.12s 0.12s 10/10
5 archil 98.4 0.13s 0.22s 0.22s 10/10
6 lightning 98.1 0.18s 0.22s 0.22s 10/10
7 beam 97.8 0.19s 0.26s 0.26s 10/10
8 blaxel 97.6 0.22s 0.27s 0.27s 10/10
9 vercel 96.7 0.29s 0.40s 0.40s 10/10
10 upstash 96.7 0.31s 0.37s 0.37s 10/10
11 superserve 96.2 0.30s 0.49s 0.49s 10/10
12 runloop 95.0 0.49s 0.52s 0.52s 10/10
13 modal 94.4 0.48s 0.68s 0.68s 10/10
14 tensorlake 93.0 0.68s 0.72s 0.72s 10/10
15 cloud-run 91.5 0.75s 1.00s 1.00s 10/10
16 tenki 91.0 0.88s 0.92s 0.92s 10/10
17 e2b 90.6 0.89s 1.02s 1.02s 10/10
18 daytona 87.1 0.79s 2.03s 2.03s 10/10
19 cloudflare 78.9 1.92s 2.38s 2.38s 10/10
20 codesandbox 73.7 2.42s 2.94s 2.94s 10/10
21 opencomputer 47.5 2.08s 27.89s 27.89s 10/10
22 hopx 0.0 0.00s 0.00s 0.00s 0/10

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.02s 0.03s 0.03s 10/10
2 declaw 99.1 0.08s 0.11s 0.11s 10/10
3 northflank 99.0 0.10s 0.10s 0.10s 10/10
4 createos 99.0 0.09s 0.12s 0.12s 10/10
5 lightning 98.4 0.16s 0.17s 0.17s 10/10
6 archil 98.3 0.16s 0.19s 0.19s 10/10
7 beam 98.1 0.16s 0.22s 0.22s 10/10
8 blaxel 97.4 0.24s 0.29s 0.29s 10/10
9 vercel 96.3 0.33s 0.44s 0.44s 10/10
10 superserve 95.9 0.39s 0.44s 0.44s 10/10
11 upstash 95.6 0.42s 0.48s 0.48s 10/10
12 runloop 94.5 0.53s 0.58s 0.58s 10/10
13 tensorlake 92.3 0.76s 0.80s 0.80s 10/10
14 modal 92.1 0.54s 1.16s 1.16s 10/10
15 cloud-run 91.9 0.62s 1.10s 1.10s 10/10
16 tenki 90.0 0.97s 1.04s 1.04s 10/10
17 daytona 88.8 0.82s 1.58s 1.58s 10/10
18 e2b 88.8 0.97s 1.35s 1.35s 10/10
19 codesandbox 68.3 2.86s 3.62s 3.62s 10/10
20 cloudflare 61.0 3.40s 4.65s 4.65s 10/10
21 opencomputer 39.6 3.40s 35.70s 35.70s 10/10
22 hopx 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
beam 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
blaxel 7/7 62.06s 2.16s 2.12s 11.49s 39.58s 1/1 OK
cloud-run 2/7 4.46s 0.33s 0.00s 0.00s 0.00s 0/1 OK
cloudflare 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 7/7 48.37s 3.19s 2.69s 11.97s 25.48s 1/1 OK
daytona 7/7 101.33s 3.75s 2.50s 15.22s 65.03s 1/1 OK
declaw 7/7 113.29s 10.40s 7.19s 31.37s 46.08s 1/1 OK
e2b 7/7 79.21s 9.43s 3.29s 15.79s 40.99s 1/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
lightning 7/7 44.66s 3.80s 2.72s 12.67s 19.08s 1/1 OK
modal 7/7 92.46s 4.23s 2.37s 14.75s 62.89s 1/1 OK
namespace 7/7 38.97s 4.82s 0.96s 7.50s 19.83s 1/1 OK
northflank -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
opencomputer 6/7 24.82s 2.86s 1.68s 16.14s 0.00s 0/1 OK
runloop 2/7 88.06s 0.00s 0.00s 22.41s 35.99s 1/1 OK
superserve 7/7 96.61s 5.91s 4.78s 23.78s 57.98s 1/1 OK
tenki 6/7 23.29s 5.50s 3.91s 9.89s 0.00s 0/1 OK
tensorlake 7/7 56.24s 10.12s 1.46s 10.30s 29.46s 1/1 OK
upstash 7/7 49.78s 3.96s 1.73s 9.57s 24.17s 1/1 OK
vercel 7/7 72.46s 16.64s 1.97s 11.52s 35.33s 1/1 OK

View full run

kisernl added a commit that referenced this pull request Jul 27, 2026
Resolves the conflicts surfaced after master advanced with the pnpm-workspace + SDK-port restructuring (#207).

Conflicts resolved:

- 3 file-location rename conflicts (scripts/* -> benchmarks/scripts/*): auto-detected by git's rename detection; the 3 new files (gen-provider-vars.ts, provider-vars.json, write-vault-envdef.mjs) landed at benchmarks/scripts/ alongside the other auto-renamed scripts.

- 4 workflow content conflicts (browser, sandbox-dax, sandbox-tti, storage): kept the new vault-load steps (Load ingest secrets from Namespace vault) over master's GH-secret env blocks, AND adopted master's pnpm + benchmarks/src/ + benchmarks/scripts/ paths.

- scripts/write-vault-envdef.mjs paths updated in all 7 workflows to benchmarks/scripts/write-vault-envdef.mjs.

- gen-provider-vars.ts: imports updated to resolve via the new benchmarks/{mode}/providers.ts layout; doc comment updated.

After re-resolve YAML still parses cleanly via js-yaml. Helper smoke-tests produce the same envdefs as before. Real-CLI roundtrip on hyperbrowser still resolves a 31-char secret end-to-end.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants